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Abstract 

Citation metrics have rapidly gained importance in today's landscape and are being increasingly utilized as a 
yardstick in making several important decisions regarding academic funding and appointments. The impact factor 
has traditionally been the metric most often employed in this regard. However, the emergence of the Hirsch index 
has provided an alternative to the impact factor. The h-index, despite its flaws, continues to gain acceptance and 
popularity in the medical community. Several medical journals have evaluated and endorsed the use of the h- 
index. However, it must be interpreted with all of its limitations in mind. 



Introduction 

Over the years, different citation indices and biblio- 
metric parameters have been formulated to measure 
academic output and scholarly activity of researchers 
and academicians across the globe. The importance of 
these metrics springs from the fact that various agencies 
bequeath grants and funds on the basis of these indices. 
Thus they influence the amount, structure and orienta- 
tion of research allocation and endorsement [1], In a 
dynamic world with limited resources [2], the impor- 
tance of such bibliometric measures cannot, therefore, 
be denied. These indices are also used as a foundation 
for conferring academic awards, hierarchal promotions, 
tenures, fellowships, salary increments, recruitments and 
leadership positions [2-5]. These indices are surrogate 
markers of scientific output [6] and ultimately demon- 
strate how "efficient" and "effective" a journal is [7]. 
They can also be considered a self-regulated effort to 
enforce some measure of quality control in a rapidly 
growing industry [6]. 

Citation metrics heavily influence the trends of sub- 
scription for various scientific journals and serve as a 
yardstick for authors when they deliberate about where 
to submit their next best scientific works [7]. According 
to Ogden et al, "the attraction of having a simple single 
number to judge complex issues is too great" [7]. Despite 
their potential shortcomings, some of these citation 
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metrics continue to garner immense momentum and 
acceptance in the scientific world. 

Impact factor 

Traditionally, the citation metric most often employed 
in medical circles to evaluate a journal's standing is the 
impact factor (IF). The IF, first proposed by Eugene 
Garfield and Irving Sher in 1963 [3], is published by 
Thomson Scientific Reuters on an annual basis in the 
Journal Citation Reports (JCR)[1]. The rationale behind 
the original derivation of IFs was based on their utility 
in the ease of selection of journals in the Science 
Citation Index (SCI) [7]. The IF of a journal for a parti- 
cular year is calculated as "a ratio of the number of 
research papers cited from that journal to the total 
number of "citable" papers it has published, using the 
time bracket of previous two years" [4]. As such, several 
factors influence the IF including, but not limited to, the 
type of article, the discipline of research and the number 
of "citable" articles a journal publishes [3]. 

Table 1 lists the potential problems with IF usage in 
the contemporary medical landscape [3,4,7-9]. The IF 
has been criticized because of its propensity for "biased 
selection" [10,11]. Although having an over-simplified 
measure such as the IF leases convenience to bioscien- 
tists, one cannot deny the fact that this situation is actu- 
ally akin to a stock exchange where major shareholders, 
or highly cited medical journals, wield a disproportion- 
ate degree of influence or monopoly on the rest of the 
investors in the industry [10]. 
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Table 1 Pitfalls of using impact factor [3,4,7-9] 



Pitfall 


Detail 


Transparency issues 


Thompson Scientific decides what is a "citable" research item. Books, book chapters and conference 




proceedings not included. 


Ambiguity in calculation 


'The definition for citing items is broader than for cited items". Also, difficult to reproduce the calculation. 


Shortcomings of 2 - year temporal 


Citation fluctuations make the inclusion of citations limited to the previous two years unreliable. 


bracket for citations 





No consideration of "citation half-life" in Citations change as the article and the journal itself matures, 
calculating IF =t= 



Over-representation 


Medical literature in English language or by a particular publisher or from a particular geographic region 




(e.g. North America) or about a particular subject (e.g. basic sciences) or of a particular type (e.g.a review) 




is disproportionately represented and cited. 


Homonymy, Synonymy 


Many authors sharing the same name or one article with many variants. Articles where author's name 




misspelled will be missed. 


Delays 


Delays in registration of citations, delays in peer-review and publication process detrimental for journals of 




disciplines with long turn-over times. 


Abridged referencing 


Many journals restrict the number of articles that can be cited by authors. Important research maybe 




potentially disregarded and not cited. 


Miscellaneous 


"Gift" authorships, self-citations and "flattery" citations 



4= IF = Impact Factor 



The h-index 

In order to circumvent these pitfalls associated with the 
usage of IF, the h-index was proposed as a bibliometric 
tool for individual authors, research groups or journals 
[7], Most frequently, the h-index is utilized as an indica- 
tor of "individual scientific achievement" [9] because a 
"book should not be judged by its cover"[12]. 

Physicist Jorge E. Hirsch at the University of Califor- 
nia, San Diego, first explained the calculation of the 
h-index in 2005 in his seminal publication as follows: "a 
scientist has the index h if h of his or her N p papers 
have at least h citations each and the remainder of the 
papers (N p - h) have < h citations each, where N p is the 
number of papers published over n years" [2]. For exam- 
ple, if a scientist has an h-index of 70, it stands to follow 
that he has published atleast 70 papers with at least 70 
citations each. Hirsch also listed some caveats that need 
to be considered when using the h-index (Table 2) [2,9]. 
Using the h-index as a guidepost, Hirsch also proposed 
the following criteria for faculty promotions at major 
research institutions, a) advancement to associate pro- 
fessor/tenure at h ~ 12, b) advancement to full professor 



at h « 18, c) fellowship or membership of major 
academic or professional societies may be granted at h > 
15 [2]. Scholars can calculate their h-index at search 
engines such as the ISI Web of Science [13], Elsevier's 
Scopus [14] or Google Scholar [15]. 

In a second paper by Hirsch in 2007, the h-index was 
shown to be a better predictor of future scientific per- 
formance and productivity when compared to other 
measures such as total number of citations, mean cita- 
tions per paper and total number of papers [9]. Addi- 
tionally, Hirsch argued that the h-index is fashioned in 
such a way that it predicts individual cumulative 
achievement better than other indices in a differential 
manner, as is the case for a paper with multiple co- 
authors with varying levels of seniority. On the other 
hand, indices such as Egghe's g index, Jin et al.'s AR 
index and Komulski's H index are inferior because they 
indiscriminately credit a highly cited paper to all its 
co-authors [9]. In contrast to the championing of the 
h-index by Hirsch, Lehmann et al [16] reported the 
superiority of the mean number of citations per paper 
over the h-index [9]. Similarly, Honekopp et al [12] 



Table 2 Caveats with the use of h-index outlined by Hirsch [2,9] 



# 


Caveats 


1. A single number such as the h-index only tells a part of the story and never the whole story. 


2. 


Researchers in non-stream fields will not achieve very high h-indices. 


3. 


Skewness in the distribution of citations possible; affects the representativeness of the h-index. 


4. 


A scientist with a few but very highly cited papers will still have a low h-index. 


5. 


Increased collaborations likely to inflate the h-value. 


6. 


Self-citations can increase the h-index. This effect is more pronounced at lower h-indices. 


7. 


Senior authors and seasoned researchers likely to have a higher h-index when compared to their junior colleagues. 



Saleem International Archives of Medicine 201 1, 4:25 
http://www.intarchmed.eom/content/4/1/25 



Page 3 of 4 



showed that the IF outperforms the h-index when pre- 
dicting the future citations of an article. 

Another issue with the h-index that merits discussion 
is the inability of the metric to differentiate between 
scientists who publish but their work goes "uncited" and 
researchers who publish little to begin with [6]. The 
same can be considered as an advantage of the h-index 
in that the authors with "practically" similar research 
profiles have similar h-indices [6]. 

Several journals have now evaluated the value of the 
h-index as a viable citation metric including Retrovirol- 
ogy [17,18], Journal of Neurosurgery [19-21], European 
Archives of Oto-Rhino-Laryngology [22], Academic 
Radiology [23], Breast Cancer Research and Treatment 
[24], Harvard Review of Psychiatry [25] and Urology 
[26]. These studies have shown that the h-index is a 
robust metric to evaluate scientific research in these 
disciplines. 

The h-index has been described as being inherently 
biased towards more seasoned researchers with long 
spanning career trajectories [27,28]. It appears to favor 
researchers who publish in fields with high citation fre- 
quencies [1]. Gender also appears to have an impact on 
the h-index via its influence on productivity [27,28]. For 
developing countries, the additional problem posed in 
the computation of the h-index is that many authors 
heavily publish in local and national journals which may 
not be tracked by or be indexed in common scientific 
search engines or databases [29]. This may distort the 
calculation of the h-index for these researchers as these 
local journals may have very limited "citability" and cir- 
culation [29]. There is thus a need to make the journals 
in developing countries more stream-lined [5]. Till that 
time, it seems unfair to compare the scientific achieve- 
ments of a scientist based in a developing country with 
those of a scientist in the developed world solely by the 
use of the h-index [5]. 

Although the h-index is considered to perform at a 
superior level when considering scientists in the same 
scientific category, there are exceptions to the rule parti- 
cularly when one considers the possibility of the exis- 
tence of subcategories within a category [5], Like the IF, 
this index also does not consider books, book chapters 
or conference proceedings in its computation [5]. The 
h-index is also not sensitive for changes in academic 
performance and it can never decrease [30]. 

An additional measure, the e-index, has also been 
introduced more recently to provide more information 
about a researcher's published works [31,32]. Dodson 
believed that the h-index underestimates the actual 
number of citations by as much as 50%[31,33]. The e- 
index refers to the "further impact" that scientists have 
over and above the h-index. This is the "excess" cita- 
tions that a researcher has in addition to those that 



have been used to compute the h-index [31,32]. To 
counter the effect of the author's age on the h-index, an 
m-score has been proposed which is calculated as "the 
h-index divided by the number of years since the first 
publication" [6,28]. In addition, a "mentoring index" has 
also been suggested [18] to give due and weighted credit 
to both senior and junior scientists who are the mentors 
and mentees respectively. 

Future directions 

There is a need to find a balance between the quantita- 
tive and qualitative aspects of bibliometric indices. Addi- 
tionally, there is a need to recognize the importance of 
expert opinion and review panels in deciding the scienti- 
fic value of an article in addition to the numeric value 
assigned through a bibliometric index [34]. Despite its 
myriad advantages, a single index cannot, for all realistic 
purposes, be the ultimate authority on the scientific 
integrity and quality of scientific research [8]. More and 
more importance is being attached to bibliometric 
indices now and we need a modus operandi that is fair, 
honest, sophisticated, up-to-date, systematic and effec- 
tive [1,10]. 

The h-index is certainly a step in the right direction. 
However, the verdict on the superiority of the h-index 
over other metrics is not unanimous at the moment, 
although positive trends can be well appreciated. This 
disputation invites further studies which may be best 
directed at the evaluation of the original h-index instead 
of derivation of further variants that will only add to the 
complexity of the problem [35]. Such studies should 
also provide a measure of concentration like the Gini 
coefficient or the Herfindahl index [35]. 

The importance of evaluating and rectifying flaws in 
current citation practices also needs to be highlighted. 
In this vein, journals can adopt the practice of random 
citation auditing [36]. Also, current bibliometric indices 
such as the h-index attach weight to frequency of cita- 
tion. However, some authors have proposed that high 
readership be taken into account instead to gauge the 
true impact of a research paper [18]. 

Conclusions 

When compared to similar bibliometric indices, the h- 
index undoubtedly represents an appealing prospect. It 
has quickly emerged to become a powerful nucleus in 
the universe of citation metrics that has captured the 
attention and fascination of researchers and scientists 
worldwide [37]. It is also being considered as a sustain- 
able alternative to the IF. Although not the quintessen- 
tial or gold-standard citation metric, the h-index still 
represents one of the most holistic and realistic indica- 
tors of a researcher's credentials. It attempts to accord 
importance to both the quantity (number of 
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publications) and quality (number of citations) of scien- 
tific research [38]. 

However, at the same time, we need to be mindful 
that like any other citation metric that maybe exces- 
sively relied upon in this climate of financial uncertainty 
[31] and increasing competition [5], it can engender a 
vicious cycle of "publish or perish", "adapt or perish", 
"citation game-playing" or "citation coalitions and net- 
works" [5,36,39,40] while also quelling creative bravado 
[41]. It must, therefore, be interpreted cautiously in the 
backdrop of all of its limitations [27,37]. At the end of 
the day, the h-index essentially remains the corollary of 
complex statistical jargon. It remains a matter of intense 
debate whether scientific talent can truly be quantified 
by a numerical entity such as the h-index [41]. 

Acknowledgements 

None. 

Authors' contributions 

TS was involved in the conception, writing and final approval of the 
manuscript. 

Competing interests 

The author declares that they have no competing interests. 

Received: 6 April 201 1 Accepted: 7 July 201 1 Published: 7 July 201 1 

References 

1. Moed HF: New developments in the use of citation analysis in research 
evaluation. Arch Immunol Ther Exp (Warsz) 2009, 57:13-8. 

2. Hirsch JE: An index to quantify an individual's scientific research output. 
Proc Natl Acad Sci USA 2005, 102:16569-72. 

3. Grzybowski A: The journal impact factor: how to interpret its true value 
and importance. Med Sci Monit 2009, 15:SRl-4. 

4. Saleem T: Impact factor - a pandora's box. J Pak Med Assoc 2009, 
59:499-500. 

5. Kellner AW, Ponciano LC: H-index in the Brazilian Academy of Sciences: 
comments and concerns. An Acad Bras Oenc 2008, 80:771-81. 

6. Sebire NJ: H-index and impact factors: assessing the clinical impact of 
researchers and specialist journals. Ultrasound Obstet Gynecol 2008, 
32:843-5. 

7. Ogden TL, Bartley DL: The ups and downs of journal impact factors. Ann 
Occup Hyg 2008, 52:73-82. 

8. Ha TC, Tan SB, Soo KC: The journal impact factor: too much of an 
impact? Ann Acad Med Singapore 2006, 35:91 1-6. 

9. Hirsch JE: Does the H index have predictive power? Proc Natl Acad Sci 
USA 2007, 104:19193-8. 

10. Oh VM: The journal numbers game. Ann Acad Med Singapore 2006, 
35:846-7. 

1 1. Cheek J, Garnham B, Quan J: What's in a number? Issues in providing 
evidence of impact and quality of research(ers). Qualitative Health Res 
2006, 16:423-35. 

12. Honekopp J, Kleber J: Sometimes the impact factor outshines the H 
index. Retrovirology 2008, 5:88. 

13. Web of Science, [http://thomsonreuters.com/products_services/science/ 
science_products/a-z/web_of_science/]. 

14. Scopus, [http://www.scopus.com/home.url]. 

1 5. Google Scholar, [http://www.scholar.google.com]. 

16. Lehmann S, Jackson AD, Lautrup BE: Measures for measures. Nature 2006, 
444:1003-4. 

1 7. Jeang KT: Impact factor, H index, peer comparisons, and Retrovirology: is 
it time to individualize citation metrics? Retrovirology 2007, 4:42. 

18. Jeang KT: H-index, mentoring-index, highly-cited and highly-accessed: 
how to evaluate scientists? Retrovirology 2008, 5:106. 



19. Lee J, Kraus KL, Couldwell WT: Use of the h index in neurosurgery. 
Clinical article. J Neurosurg 2009, 1 1 1 :387-92. 

20. Spearman CM, Quigley (V1J, Quigley MR, Wilberger JE: Survey of the h 
index for all of academic neurosurgery: another power-law 
phenomenon? J Neurosurg 2010, 1 13:929-33. 

21. Ponce FA, Lozano AM: Academic impact and rankings of American and 
Canadian neurosurgical departments as assessed using the h index. J 
Neurosurg 2010, 113:447-57. 

22. Kulasegarah J, Fenton JE: Comparison of the h index with standard 
bibliometric indicators to rank influential otolaryngologists in Europe 
and North America. Eur Arch Otorhinolaryngol 2010, 267:455-8. 

23. Rad AE, Brinjikji W, Cloft HJ, Kallmes DF: The H-index in academic 
radiology. Acad Radiol 2010, 17:817-21. 

24. Healy NA, Glynn RW, Scutaru C, Groneberg D, Kerin MJ, Sweeney KJ: The h 
index and the identification of global benchmarks for breast cancer 
research output. Breast Cancer Res Treat 201 1. 

25. Hunt GE, Cleary M, Walter G: Psychiatry and the Hirsch h-index: The 
relationship between journal impact factors and accrued citations, hiarv 
Rev Psychiatry 2010, 18:207-19. 

26. Benway BM, Kalidas P, Cabello JM, Bhayani SB: Does citation analysis 
reveal association between h-index and academic rank in urology? 
Urology 2009, 74:30-3. 

27. Kelly CD, Jennions MD: H-index: age and sex make it unreliable. Nature 

2007, 449:403. 

28. Kelly CD, Jennions MD: The h index and career assessment by numbers. 
Trends Ecol Evol 2006, 21:167-70. 

29. (vlishra DC: Citations: rankings weigh against developing nations. Nature 

2008, 451:244. 

30. Rousseau R: New developments related to the Hirsch index. Science Tocus 
2006, 1:23-25. 

31. Dodson MV: Citation analysis: Maintenance of h-index and use of e- 
index. Biochem Biophys Res Commun 2009, 387:625-6. 

32. Zhang CT: The e-index, complementing the h-index for excess citations. 
PtoS One 2009, 4:e5429. 

33. Thomaz PG, Assad RS, Moreira LF: Using the impact factor and H index to 
assess researchers and publications. Arg Bras Cardiol 2011, 96:90-93. 

34. Allen L, Jones C, Dolby K, Lynn D, Walport M: Looking for landmarks: the 
role of expert review and bibliometric analysis in evaluating scientific 
publication outputs. PLoS One 2009, 4:e5910. 

35. Bornmann L, Daniel HD: The state of h index research. Is the h index the 
ideal way to measure research performance? EMBO Rep 2009, 10:2-6. 

36. Todd PA, Ladle RJ: Citations: poor practices by authors reduce their 
value. Nature 2008, 451:244. 

37. Ball P: Achievement index climbs the ranks. Nature 2007, 448:737. 

38. Balandin S, Stancliffe RJ: Impact factors and the h-index: what researchers 
and readers need to know. Augment Altern Commun 2009, 25:1-3. 

39. Williamson JR: My h-index turns 40: my midlife crisis of impact. ACS 
Chem Biol 2009, 4:311-3. 

40. Purvis A: The h index: playing the numbers game. Trends Ecol Evol 2006, 
21:422. 

41. Kotov NA: Fraud, the h-index, and Pasternak. ACS Nano 2010, 4:585-6. 



doi:1 0.1 1 86/1 755-7682-4-25 

Cite this article as: Saleem: The Hirsch index - a play on numbers or a 
true appraisal of academic output? International Archives of Medicine 201 1 
4:25. 



